Review

The concept of likelihood and maximum likelihood estimation (MLE) have been at the core of much of statistical modeling for about 100 years

In 21st Century, likelihood and MLE ideas continue to be foundational
- The principles we can easily visualize in a few dimensions apply in high dimenstions
- An understanding of these principles helps you understand modern large scale methods
Understanding the concept of likelihood and the use of MLE methods is key to understanding many parametric statistical methods
Likelihood is a measure of how well a model fits data
MLE is a generic methods for parameter estimation
MLE used widely for machine learning models, including some deep learning models

Review

Statistical inference seeks to characterize the uncertainty in statistical point estimates

Statistics are estimates of population parameters
Inferences using statistics must consider the uncertainty in the estimates
Confidence intervals quantify uncertainty in statistical estimates
- Two-sided confidence intervals: express confidence that a value is within some range around the point estimate
- One-sided confidence intervals: express confidence that the point estimate is greater or less than some range of values

Review

Nonparametric bootstrap estimation is widely useful and requires minimal assumption

Bootstrap distribution is comprised of values of the statistic computed from bootstrap resamples of the original observations (data sample)
Computing bootstrap distribution requires no assumptions about population distribution!
- Bootstrap resampling substitutes computer power for paper and pencil statistician power
Bootstrap resampling estimates the bootstrap distribution of a statistic
- Compute mostly likely point estimate of the statistic, or bootstrap estimate
- The bootstrap confidence interval is computed from the bootstrap distribution

Review

There are several variations of the basic nonparametric bootstrap algorithm

One sample bootstrap
- Inference on single statistic
Two sample bootstrap
- Inference on different statistic
Special cases
- Correlation coefficients - part of your assignment

Review

Re-sampling methods are general and powerful but, there is no magic involved! There are pitfalls!

If a sample is biased, the re-sampled statistic estimate based on that sample will be biased
- Results can be no better than the sample you start with
- Example; the bootstrap estimate of mean is the unbiased sample estimate, $\bar{x}$, not the population parameter, $\mu$
The sample variance and Cis can be no better than the sample distribution allows
- Be suspicious of overly optimistic confidence intervals
- CIs can be optimistically biased
Are computationally intensive, but often highly parallelizable

Introduction Baysian Models

Despite the long history, Bayesian models have not been used extensively until recently

Two traditions in statistics
- Frequentist we have been working with previously
- Bayesian statistics
Limited use is a result of several difficulties
- Rarely taught for much of the 20th Century
- The need to specify a prior distribution has proved a formidable intellectual obstacle
- Modern Bayesian methods are often computationally intensive and have become practical only with cheap computing
Recent emergence of improved software and algorithms has resulted in wide and practical access to Bayesian methods

Introduction

Bayesian analysis is a contrast to frequentist methods

The objective of Bayesian analysis is to compute a posterior distribution
Contrasts with frequentist statistics is to compute a point estimate and confidence interval from a sample
Bayesian models allow expressing prior information in the form of a prior distribution
Selection of prior distributions can be performed in a number of ways
The posterior distribution is said to quantify our current belief
We update beliefs based on additional data or evidence
A critical difference with frequentist models which must be computed from a complete sample
Inference can be performed on the posterior distribution by finding the maximum a postiori (MAP) value and a credible interval

Bayesian Model Use Case

Bayesian methods made global headlines with the successful location of the missing Air France Flight 447

Aircraft had disappeared in little traveled area of the South Atlantic Ocean
Conventional location methods had failed to locate the wreckage; potential search area too large
Bayesian methods rapidly narrowed the prospective search area
- Used ‘prior information’ on aircraft heading and time of sattelite transmisison

$Posterior distribution of locations of Air France 447$

Posterior distribution of locations of Air France 447

Bayesian vs. Frequentist Views

With greater computational power and general acceptance, Bayes methods are now widely used

Among pragmatists
- Some problems are better handled by frequentist methods
- Some problems with Bayesian methods
Bayes models allow us to express prior information
Models that fall between these extremes are also in common use
- Methods include the so-called empirical Bayes methods.

Bayesian vs. Frequentist Views

Can compare the contrasting frequentist and Bayesian approaches

Comparison of frequentist and Bayes methods

Review of Bayes Theorem

Bayes’ Theorem is fundamental to Bayesian data analysis.

Start with:

\[P(A \cap B) = P(A|B) P(B) \]

We can also write:

\[P(A \cap B) = P(B|A) P(A) \]

Eliminating $P(A \cap B):$

\[ P(B)P(A|B) = P(A)P(B|A)\]

And finally, Bayes theorem!

\[P(A|B) = \frac{P(B|A)P(A)}{P(B)}\]

Bayes Theorem

Bayes Theorem!

Marginal Distributions

In many cases we are interested in the marginal distribution

Example, it is often the case that only one or a few parameters of a joint distribution will be of interest
- In other words, we are interested in the marginal distribution of these parameters
- The denominator of Bayes theorem, $P(data)$, can be computed as a marginal distribution
Consider a multivariate probability density function with $n$ variables, $p(\theta_1, \theta_2, \ldots, \theta_n)$
- Marginal distribution is the distribution of one variable with the others integrated out.
- Integrate over all other variables $\{ \theta_2, \ldots, \theta_n \}$ the result is the marginal distribution, $p(\theta_1)$:

\[p(\theta_1) = \int_{\theta_2, \ldots, \theta_n} p(\theta_1, \theta_2, \ldots, \theta_n)\ d\theta2, \ldots, d \theta_n\] - But computing this integral is not easy!

Marginal Distributions

For discrete distributions compute the marginal by summation
Or, for discrete samples of a continuous distribution
Example, we know posterior distribution of a parameter $\theta$ but really want marginal distribution of the parameter value

\[ p(\theta) = \sum_{x \in \mathbf{X}} p(\theta |\mathbf{X})\ p(\mathbf{X}) \]

Now we have the marginal distribution of $\theta$
Or, we need to find the denominator for Bayes theorem to normalize our posterior distribution:

\[ p(\mathbf{X}) = \sum_{\theta \in \Theta} p(\mathbf{X} |\theta) p(\theta) \]

We can compute $p(\mathbf{X})$ from samples without ever solving the integral directly!

Interpreting Bayes Theorem

How can you interpret Bayes’ Theorem?

For model parameter estimation problem:

\[Posterior\ Distribution = \frac{Likelihood \bullet Prior\ Distribution}{Evidence} \]

Or, Bayes’ theorem in terms of model parameters:

\[ posterior\ distribution(𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑒𝑟𝑠│𝑑𝑎𝑡𝑎) = \\ \frac{Likelihood(𝑑𝑎𝑡𝑎|𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑒𝑟𝑠)\ 𝑃rior(𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑒𝑟𝑠)}{P(data)} \]

Summarized as:

\[ 𝑃(𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑒𝑟𝑠│𝑑𝑎𝑡𝑎) = \frac{P(𝑑𝑎𝑡𝑎|𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑒𝑟𝑠)\ 𝑃(𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑒𝑟𝑠)}{P(data)} \]

Interpreting Bayes Theorem

What do these terms actually mean?

Posterior distribution of the parameters given the evidence or data, the goal of Bayesian analysis
Prior distribution is chosen to express information available about the model parameters apriori
Likelihood is the conditional distribution of the data given the model parameters
Data or evidence is the distribution of the data and normalizes the posterior

Relationships can apply to the parameters in a model; partial slopes, intercept, error distributions, lasso constants, etc

Applying Bayes Theorem

We need a tractable formulation of Bayes Theorem for computational problems

We must avoid directly summing all of the possibilities to compute the denominator, $P(B)$
- In many cases, computing this denominator directly is intractable
Some interesting facts about conditional probabilities:

\[ 𝑃(𝐵 \cap A) = 𝑃(𝐵|𝐴)𝑃(𝐴) \\ And \\ 𝑃(𝐵)=𝑃(𝐵 \cap 𝐴)+𝑃(𝐵 \cap \bar{𝐴}) \]

Where, $\bar{A} = not\ A$, and the marginal distribution, $P(B)$, can be written:

\[ 𝑃(𝐵)=𝑃(𝐵|𝐴)𝑃(𝐴)+𝑃(𝐵│ \bar{𝐴})𝑃(\bar{𝐴}) \]

Applying Bayes Theorem

Using the foregoing relations we can rewrite Bayes Theorem as:

\[ P(A|B) = \frac{P(A)P(B|A)}{𝑃(𝐵│𝐴)𝑃(𝐴)+𝑃(𝐵│ \bar{𝐴})𝑃(\bar{𝐴})} \\ \]

Computing the denominator requires summing all cases in the subsets $A$ and $not\ A$
- This is a bit of a mess!
- And, the siguation is worse if there are multiple alternative hypotheses!
- Fortunately, we can often avoid computing this denominator by force

Eewrite Bayes Theorem as:

\[𝑃(𝐴│𝐵)=𝑘∙𝑃(𝐵|𝐴)𝑃(𝐴)\]

Ignoring the normalization constant $k$:

\[𝑃(𝐴│𝐵) \propto 𝑃(𝐵|𝐴)𝑃(𝐴)\]

Interpreting Bayes Theorem

Denominator must account for all possible outcomes, or alternative hypotheses, $h'$:

\[Posterior(hypothesis\ |\ evidence) =\\ \frac{Likelihood(evidence\ |\ hypothesis)\ prior(hypothesis)}{\sum_{ h' \in\ All\ possible\ hypotheses}Likelihood(evidence\ |\ h')\ prior(h')}\]

This is a formidable problem!

Bayes Theorem Example

Hemophilia is a serious genetic condition expressed on any X chromosome

Women have two X chromosomes and are unlikely to exhibit hemophilia
- One X chromosome inherited from each parent
- Must inherit hemophilia from both parents
Men have one X chromosome and one Y chromosome
- Inherit Y chromosome from the father
- Inherit X chromosome, and possibly hemophilia, from the mother
Say a woman has a brother who exhibits hemophilia
- Y chromosome expression is $\theta = 1.0$
- Woman’s father does not exhibit hemophilia, $\theta = 0$
- Our prior probability that she carries the genetic marker for hemophilia $\theta = 0.5$

Bayes Theorem Example

As evidence the woman has two sons (not identical twins) with no expression of hemophilia

What is the likelihood for the two sons $X = (x_1,x_2)$ not having hemophilia?
Two possible cases here
- Case where woman caries one X chromosome with hemophilia expression, $\theta = 1$
- Case where woman does not carry an X chromosome with hemophilia expression, $\theta = 0$

\[ p(x_1=0, x_2=0 | \theta = 1) = 0.5 * 0.5 = 0.25 \\ p(x_1=0, x_2=0 | \theta = 0) = 1.0 * 1.0 = 1.0 \]

Note: we are neglecting the possibility of a mutations in one of the sons

Bayes Theorem Example

Use Bayes theorem to compute probability woman carries an X chromosome with hemophilia expression, $\theta = 1$

\[ p(\theta=1 | X) = \frac{p(X|\theta=1) p(\theta=1)}{p(X|\theta=1) p(\theta=1) + p(X|\theta=0) p(\theta=0)} \\ = \frac{0.25 * 0.5}{0.25 * 0.5 + 1.0 * 0.5} = 0.20 \]

The evidence of two sons without hemophilia causes us to update our belief that the probability of the woman carrying the disease

Note: The denominator is the sum over all possible hypthises, the marginal distribution of the observations $\mathbf{X}$

Simplified Relationship for Bayes Theorem

How to we interpret the foregoing relationship?

Consider the following relationship:

\[Posterior\ Distribution \propto Likelihood \bullet Prior\ Distribution \\ Or\\ 𝑃( 𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑒𝑟𝑠 │ 𝑑𝑎𝑡𝑎 ) \propto 𝑃( 𝑑𝑎𝑡𝑎 | 𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑒𝑟𝑠 )𝑃( 𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑒𝑟𝑠 ) \]

We can find an unnormalized function proportional to the posterior distribution
Sum over the function to find the marginal distribution $P(B)$
Approach can transform an intractable computation into a simple summation

Creating Bayes models

The goal of a Bayesian analysis is computing and performing inference on the posterior distribution of the model parameters

The general steps are as follows:

Identify data relevant to the research question
Define a sampling plan for the data. Data need not be collected in a single batch
Define the model and the likelihood function; e.g. regression model with Normal likelihood
Specify a prior distribution of the model parameters
Use the Bayesian inference formula to compute posterior distribution of the model parameters
Update the posterior as data is observed
Inference on the posterior can be performed; compute credible intervals
Optionally, simulate data values from realizations of the posterior distribution. These values are predictions from the model.

Updating Bayesian Models

An advantage of Bayesain model is that it can be updated as new observations are made

In contrast, for frequentist models data must be collected completely in advance
We update our belief by adding new evidence
The posterior of a Bayesian model with no evidence is the prior
The previous posterior serves as a prior for model updates

How can you choose a prior?

The choice of the prior is a difficult, and potentially vexing, problem when performing Bayesian analysis

The need to choose a prior has often been cited as a reason why Bayesian models are impractical
General guidance is that a prior must be convincing to a skeptical audience is vague in practice
Some possible approaches include:
Use prior empirical information
Apply domain knowledge to determine a reasonable distribution
- Use information from prior work
- For example, the viable range of parameter values could be computed from physical principles
- For example, it could be well know that there are price range for some asset
If there is poor prior knowledge for the problem a non-informative prior can be used
- One possibility is a Uniform distribution. But be careful! since a uniform prior is informative because of limits on the values!
- Other options include the Jefferys’ prior.

How can you choose a prior?

How to use prior empirical information to estimate the parameters of the prior distribution

Deriving a prior distribution in this manner is sometimes called empirical Bayes
- Has become more practical with large modern data sets
- Method somewhere between Bayesian and frequentist
- Empirical Bayes approach is often applied in practice
- Some Bayesian theoreticians do not consider this a Bayesian approach at all
Example, need a prior distribution of home prices per square foot by location
- Use pooled information to compute distribution of prices for all locations
- Use the prior with specific observations by locations to compute posteriors by location
Sometimes, a less informative prior distribution is used than the actual empirical distribution so the model is not overly constrained

Conjugate Prior Distributions

An analytically and computationally simple choice for a prior distribution family is a conjugate prior

When a likelihood function is multiplied by its conjugate distribution the posterior distribution will be in the same family as the conjugate prior
Attractive idea for cases where the conjugate distribution exists
- Analytic results can be computed
- The posterior is a known distribution
But there are many practical cases where a conjugate prior is not used
- We will address more gneral methods later

Conjugate Prior Distributions

Most commonly used distributions have conjugates, with a few examples:

Likelihood	Conjugate
Binomial	Beta
Bernoulli	Beta
Poisson	Gamma
Categorical	Dirichlet
Normal - mean	Normal
Normal - variance	Inverse Gamma
Normal - inverse variance, $\tau$	Gamma

Example using Conjugate Distribution

We are interested in analyzing the incidence of distracted drivers

Randomly sample the behavior of 10 drivers at an intersection and determine if they exhibit distracted driving or not
Data are Binomially distributed, a driver is distracted or not, with likelihood:

\[ P(k) = \binom{n}{k} \cdot \theta^k(1-\theta)^{n-k}\]

Binomial likelihood has one parameter we need to estimate, $\theta$, the probability of success

Our process is the:

Use the conjugate prior, the Beta distribution with parameters $\alpha$ and $\beta$
Using the data sample, compute the likelihood
Compute the posterior distribution of distracted driving
Add more evidence (data) and update the posterior distribution.

Example using Conjugate Distribution

What are the properties of the Beta distribution?

Beta distribution for different parameter values

Example using Conjugate Distribution

Consider the product of a Binomial likelihood and a Beta prior

Define the evidence as $n$ trials with $z$ successes
Prior is a Beta distribution with parameters $a$ and $b$, or the vector $\theta = (a,b)$
From Bayes Theorem the distribution of the posterior:

\[\begin{align} posterior(\theta | z, n) &= \frac{likelihood(z,n | \theta)\ prior(\theta)}{data\ distribution (z,n)} \\ p(\theta | z, n) &= \frac{Binomial(z,n | \theta)\ Beta(\theta)}{p(z,n)} \\ &= Beta(z + a -1,\ n-z+b-1) \end{align}\]

Example using Conjugate Distribution

There are some useful insights you can gain from this relationship:

\[ posterior(\theta | z, n) = Beta(z + a -1,\ n-z+b-1) \]

Posterior distribution is in the Beta family, as a result of conjugacy
Parameters $a$ and $b$ are determined by the prior and the evidence
Parameters of the prior can be interpreted as pseudo counts of successes, $a = pseudo\ success + 1$ and failures, $b = pseudo\ failure + 1$
- Be careful when creating a prior to add 1 to the successes and failures
- The larger the total pseudo counts, $a + b$, the stronger the prior information

-Evidence is also in the form (actual) counts of successes, $z$ and failure, $n-z$
- The more evidence the greater the influence on the posterior distribution
- Large amount of evidence will overwhelm the prior
- With large amount of evidence, posterior converges to frequentist model

Example using Conjugate Distribution

Consider example with:
- Prior pseudo counts $[1,9]$, successes $a = 1 + 1$ and failures, $b = 9 + 1$
- Evidence, successes $= 10$ and failures, $= 30$
- Posterior is $Beta(10 + 2 -1,\ 40 - 10 + 10 -1) = Beta(11,\ 39)$

Prior, likelihood and posterior for distracted driving

Sampling the Posterior

How can we find an estimate of the poster distribution?

We can sample from the analytic solution - if we have a conjugate
We can sample the likelihood and prior, take the product and normalize - for any posterior
Grid sample or Markov chain Monte Carlo (MCMC) sample

Sampling the Posterior

Grid sampling is a naive approach

Compute the probability at each point on a regular gird
- Sample over range of interesting values for variables
- Posterior if conjugate prior
- Prior and likelihood
In principle can work for any number of dimensions
- In 1-dimension is just regularly spaced points on a line
- Poor scaling to higher dimensions

Sampling grid for bivariate distribution

Sampling the Posterior

Algorithm for grid sampling to compute posterior from likelihood and prior

Procedure CreateGrid(variables, lower_limits, upper_limits): 
    # Build the sampling grid 
    return sampling_grid   
    
Procedure SampleLikelihood(sampling_value, observation_values):    
    return likelihood_function(sampling_value, observation_values)
    
Procedure Prior(sampling_values, prior_parameter_value):    
    return prior_density_function(sampling_value, prior_parameter_values)    
    
ComputePosterior(variables, lower_limits, upper_limits):    
    # Initialize the sampling grid
    Grid = CreateGrid(variables, lower_limits, upper_limits)
    
    # Initialize array to hold sampled posterior values       
    array posterior[range(Grid)]

    # Compute posterior at each sampling value in the grid  
    for sampling_value in range(lower_limits, upper_limits):   
        likelihood = SampleLikelihood(sampling_value, observation_values)
        prior = Prior(sampling_values, prior_parameter_value)   
        posterior[sampling_value] = likelihood * prior

    # Normalize the posterior       
    probability_data = sum(posterior[range(Grid)])
    posterior = posterior[range(Grid)]/probability_data 
    return posterior

Credible Intervals

How can we specify the uncertainty for a Bayesian parameter estimate?

For frequentist analysis we use confidence intervals, but not entirely appropriate
- Confidence intervals are based on a sampling distribution
- The upper and lower confidence intervals quantiles of the sampling distribution
- Bayesian analysis has no sampling distribution uses a prior distribution and likelihood
For Bayesian analysis inference performed on posterior distribution
- We use a concept known as the credible interval
- A credible interval is an interval on the Bayesian posterior distribution with the highest $\alpha$ proportion of posterior probability
Example, the $\alpha = 0.90$ credible interval encompasses the 90% of the posterior distribution with the highest density
The credible interval is sometime called the highest density interval (HDI), or highest posterior density interval (HPDI)
- These names make sense, since we seek the the densest posterior interval containing $\alpha$ probability
For symmetric distributions the credible interval can be numerically the same as the confidence interval
- In general, these two quantities can be quite different

Credible Intervals

What are the 95% credible intervals for $Beta(11,\ 39)$?

Probability of distract drivers for next 10 cars

Credible Intervals are not Confidence Intervals

How are credible intervals different from the more familiar confidence intervals?

Confidence intervials and credible intervals are conceptually quite different

A confidence interval is a purely frequentest concept
- Is an interval on the sampling distribution where repeated samples of a statistic are expected with probability $= \alpha$
- Cannot interpret a confidence interval as an interval on a probability distribution of the value of a statistic!

Credible interval is an interval on a posterior distribution of the statistic
- Credible interval is exactly what the misinterpretation of the confidence interval tries to be
- Credible interval is the interval with highest $\alpha$ probability for the statistic being estimated

For symmetric posterior distributions, the credible interval will be numerically the same as the confidence interval
- This need not be the case in general

Credible Intervals are not Confidence Intervals

Compare confidence interval and credible interval for the case of 10 observations

Credible intervals cross the density function at exactly the same density
Confidence intervals have the same CDF in the tails beyond the interval

Difference between credible and confidence intervals

Simulating from the posterior distribution: predictions

What else can we do with a Bayesian posterior distribution beyond credible intervals?

Perform simulations and make predictions
Predictions are computed by simulating from the posterior distribution
Results of these simulations are useful for several purposes, including:
- Predicting posterior values
- Model checking by comparing simulation results agree (or not) with observations

Simulating from the posterior distribution: predictions

Example; What are the probabilities of distracted drivers for the next 10 cars with posterior, $Beta(11,\ 39)$?

Probability of distract drivers for next 10 cars

Summary

Bayesian analysis is a contrast to frequentist methods

The objective of Bayesian analysis is to compute a posterior distribution
Contrasts with frequentist statistics is to compute a point estimate and confidence interval from a sample
Bayesian models allow expressing prior information in the form of a prior distribution
Selection of prior distributions can be performed in a number of ways
The posterior distribution is said to quantify our current belief
We update beliefs based on additional data or evidence
A critical difference with frequentist models which must be computed from a complete sample
Inference can be performed on the posterior distribution by finding the maximum a postiori (MAP) value and a credible interval
Predictions are made by simulating from the posterior distribution a

Summary

Bayesian analysis is in contrast to frequentist methods

The objective of Bayesian analysis is to compute a posterior distribution
Frequentist statistics seeks to compute a point estimate and confidence interval from a sample
Bayesian models allow expressing prior information in the form of a prior distribution
Selection of prior distributions can be performed in a number of ways
The posterior distribution is said to quantify our current belief
We update beliefs based on additional data or evidence
A critical difference with frequentist models which must be computed from a complete sample
Inference can be performed on the posterior distribution by finding the maximum a postiori (MAP) value and a credible interval

Introduction to Bayesian Models

Review

Review

Review

Review

Review

Introduction Baysian Models

Introduction

Bayesian Model Use Case

Bayesian vs. Frequentist Views

Bayesian vs. Frequentist Views

Review of Bayes Theorem

Bayes Theorem

Marginal Distributions

Marginal Distributions

Interpreting Bayes Theorem

Interpreting Bayes Theorem

Applying Bayes Theorem

Applying Bayes Theorem

Interpreting Bayes Theorem

Bayes Theorem Example

Bayes Theorem Example

Bayes Theorem Example

Simplified Relationship for Bayes Theorem

Creating Bayes models

Updating Bayesian Models

How can you choose a prior?

How can you choose a prior?

Conjugate Prior Distributions

Conjugate Prior Distributions

Example using Conjugate Distribution

Example using Conjugate Distribution

Example using Conjugate Distribution

Example using Conjugate Distribution

Example using Conjugate Distribution

Sampling the Posterior

Sampling the Posterior

Sampling the Posterior

Credible Intervals

Credible Intervals

Credible Intervals are not Confidence Intervals

Credible Intervals are not Confidence Intervals

Simulating from the posterior distribution: predictions

Simulating from the posterior distribution: predictions

Summary

Summary